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Abstract: We are getting more and more familiar with virtual worlds and realities. 
From social media to being forced to use a QR code for accessing the menu of a restaurant. 
This change that is happening now pushes the artists to rethink the way they are doing projects 
and to create new conventions. Because social media and QR codes requests a participatory 
involvement, the art pieces should start to have this kind of quality as well. The new 
generation of spectators is less familiarised with the classical conventions where they would 
have to stay passive and silent. There are many applications and platforms that can be used 
for creating more engaging and immersive projects. The choice of which kind of technology 
should be used it depends on the project and the type of interaction that is wanted. We can 
choose from motion capture systems, mobile apps or different devices that can provide 
different features. Even the headphones can be a tool that can be used for creating an 
immersive experience. The next step is to choose on what platforms we are going to create 
the interaction. There are many options here as well and depends also on the direction that the 
project has but also on the knowledge that we have, because many platforms can be used to 
provide almost the same tools. The one that I am focusing on is Unreal Engine, which is an 
open source game. This platform also recently released a new feature which is called 
Metahuman. With this feature we can create avatars that can be animated in real time through 
an application on the phone. The application makes the camera of our smart phone a motion 
capture device that animates the face and the head of the avatar. This can be made by either 
the performers or the spectators. 
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Introduction 

We are becoming increasingly familiar with virtual worlds and virtual reality. 
From using social media platforms to having to read a QR code to access the menu in 
a restaurant. These new conventions that are now part of our lives must also be felt in 
art and how it creates new conventions. Because social media and QR codes require 
participation, even art projects should have this quality. Young audiences are less 
familiar with classical theatrical conventions, where they have to remain passive and 
silent. 
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There are several reasons why performance or art in general is increasingly 
embracing the use of technology. The first reason is the development of society which 
forces people to use technological alternatives for logistical reasons or for the 
increased convenience they offer. There are places where this development is only 
hinted at, but there is also the opposite case where we cannot have certain facilities or 
services if we do not have a smartphone and an internet connection. Something that 
was once a luxury is now commonplace or an obligo. Another reason is the fascination 
of the possibilities that these new technologies offer artists and they choose to use 
these new tools to create new images and aesthetics, but more importantly to vary 
their narrative. The third reason or perspective that pushes art to use technology is the 
consumer of the technology itself - the viewer. 


The Spectator in the 21st Century 

The spectator is part of an evolving society, and with this evolution his reality 
changes, which will slowly make him have different needs and values. *(t. At the 
same time, spectators belong to different generations, resulting in different degrees to 
which they have been exposed to certain conventions. This exposure has made 
viewers familiar with certain conventions, an important factor being the age at which 
they first encountered those conventions. 

When we refer to conventions, we are referring both to how a person buys 
food or any other products or how they buy tickets for various means of transport and 
then travel. But the conventions that will most influence how a man builds his artistic 
values is how he spends his free time, what a man does when he has a free window or 
after he finishes work and has some time to indulge himself. Lately these times have 
started to be more and more often filled with social media activities or surfing various 
platforms on the internet. 

Long internet surfing accustoms the person or future viewer to be present in 
an environment where they are continuously making decisions or always have the 
potential to do so. Even if we just scroll on Facebook, Instagram, Tik Tok, etc. 
platforms we can always have a reaction to these contents, either through a simple 
like or a heart or a comment. Even the ability to linger longer on a photo or video is 
an interactivity capability. This convention also brings with it an exposure to the 
screen and video material. 

In addition to the use of social networks, the use of platforms such as Netflix 
or HBO GO has increasingly increased. These platforms offer the ability for the user 
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to access a wide range of films and series without the constraints of television. These 
platforms are gaining a lot of ground in popularity especially with the advent of the 
pandemic. Once again we encounter a convention that is a little different from the one 
that was popular a few years ago - television, and which brings with it an added 
interactivity. 

All these new media that we encounter every day create the need for shows to 
be interactive. The show must contain this interactive quality and possibly take it to 
another level. "The spectator's perceptive experience is due to the subjective 
capacities of his own body and nervous system. This is part of a change in the way a 
performance is perceived by spectators, and this must now be understood in terms of 
the conditions of creation and the mode of cultural reception in the 21st century... 
More is needed. The spectator wants to be more actively involved, to play a role or a 
significant part in the reception of the work. "Like the changes made by the food we 
eat, similarly we are modified and influenced by the technology of today. 


Technologies that can facilitate the creation of interactive performances 

There are many ways in which performances can become interactive. One way 
to create interactivity is through the use of technology, specifically certain 
technologies that can provide this opportunity. I will list some of these technologies 
and describe or give examples of how they can be applied. Technologies that lend 
themselves to this are: motion capture, OSC protocol and MIDI protocol, Arduino or 
any kind of development board such as Rasberry Pi, Orange Pi, Banana Pi etc. 

Each of these technologies needs to be synchronised by various methods with 
video or sound content. One or more of these can be used for each artistic creation. 
For a clearer understanding of these technologies I will detail each one separately. 


Motion capture 

Motion capture is the process of recording the movement of objects or bodies. 
This technology has a wide range of applications, being used in the military, medical 
and sports fields, and later by companies using it in gaming. There are many types of 
motion capture. They are either optical or non-optical, meaning inertial, magnetic or 
mechanical. Optical systems work by capturing physical markings, LED lights, 
reflectors, sensors or just colour. There are many companies that 
produce motion capture systems. Originally these systems were in the form of 
installations that were meant to be fixed in a room. These systems consisted of several 
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cameras fixed at the end of the room that perceived the movement of passive or active 
sensors. These complex installations are still used, they offer higher quality, but in the 
meantime simpler, portable systems have also appeared. 


Kinect 

The Xbox company launched a motion capture system back in 2010 that has 
been very popular in the arts. "The Kinect sensor has an RGB camera, a depth sensor 
consisting of an infrared laser projector, an infrared CMOS sensor and a multi-array 
microphone that allows acoustic source localization and ambient noise suppression. It 
also contains an LED light, a three-axis accelerometer and a small servo motor that 
controls the tilt of the device." 

The motto used in the marketing strategy was: you are the controller. The 
product made it into the Guinness Book of Records as the best-selling electronic 
device in history. It went on to sell 8 million units in the first 60 days of its launch. 

Initially this product did not come with a set of drivers to allow it to be used 
for various applications. Eventually, in 2011, Microsoft offered the Kinect SDK on a 
non-commercial basis, and then in 2012 it was offered commercially with the Kinect 
device designed specifically for Windows, the Xbox One. 

In 2013 the Kinect sensor version 2 is released, which has a different 
technology than version 1, being a Time-Of-Flight (TOF) camera. The basic operating 
principle is that of continuous wave sensors, a string of transmitters sends a modulated 
signal that travels to measured points, are reflected and are received by the sensor. 
The sensor acquires a 512 x 424 depth map and a 1920 x 1080 colour image at 15 to 
30 fps, depending on the lighting condition, as the sensor uses an automatic exposure 
algorithm. 

After this point, a lot of software for the Kinect sensor started to appear, and 

it will be used in a lot of areas. The area in which it has been used most optimally is 
visuals. Given the presence of visuals in theatre and dance performances, the Kinect 
began to be used in the arts as well, while opening up new perspectives on how these 
images could be created, and most importantly that these images could become 
interactive, both for the viewer and the actors or dancers. 
"Most games designed for Kinect do not promote cognitive and emotional investment 
in the viewer. The result is that Kinect software has rarely taken advantage of the 
feature that most distinguishes Kinect from other gaming peripherals: its ability to 
turn games into a theatrical event." 
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It's been used quite a bit in dance, with choreographers interested in being able 
to capture their movements and use it for various things. One software that has been 
used in such projects is MotionDraw. It is a tool designed specifically for Kinect that 
enhances the performance of images on projection. Images can be created directly 
from movements made by performers or even the audience. 

Subsequently, more complex ways of integrating this or any other motion 
capture system have emerged thanks to gaming engines such as Unity or Unreal 
Engine. These can be used to find creative ways of modifying or influencing images. 


Leap Motion 

Another piece of equipment that is very useful for interactive applications is 
Leap Motion. It is a motion capture system that optimally captures hand and finger 
movements. This device was made to be able to navigate through the computer using 
hands free without using the classic mouse. Leap Motion can be used mainly for its 
ability to recognise gestures, which can be turned into signals. It can recognise 
gestures made with one hand as well as the other, each of which can be set as a 
different signal. In another reason why he is a good option is his very small size, he 
can be hidden or masked, thus creating a little magic by having some hand movements 
in the air make changes or give concrete signals. 

This motion capture system can also be used for interactions in the virtual 
world. In 2016, in response to developments in virtual reality, the company is 
releasing software to capture hand movements in virtual space. The company began 
collaborating with many companies developing virtual reality projects. It has long 
collaborated with the likes of Asus and was used by the company OSVR which was 
producing VR headsets. They also tried a collaboration with Oculus for a medical 
project for visually impaired people. 


OSC (Open Sound Control) and MIDI (Musical Instrument Digital Interface) 
protocol 


These two protocols help us to transmit commands from one computer to 
another or from different devices to the computer, and the computer will further 
transmit the signals to projection, lights or sound. Originally each of the oranges was 
created for musical projects, but at its core each is just a signal in a computer- 
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recognisable language that can be converted into any kind of command. With each of 
the two protocols we can achieve the same results, but each has different 
characteristics. 

MIDI is an older protocol and generally devices that transmit such signals will 
need to be wired directly. The advantage is that there are already many such devices 
that can be purchased and will transmit signals without prior programming. Of course 
very many of these devices were created for music themselves, calling themselves 
MIDI controllers. These devices can be placed in the space where the audience has 
access and they can use them to change the lights, projections or sound of the 
performance. 

The thing about the OSC protocol is that it works over a local network, so we 
can connect multiple devices, be they phones, using a wi-fi router. This router does 
not need to be connected to the internet, simply connecting to this network will be 
enough for the signals transmitted via the OSC protocol to be received. To transmit 
OSC signals we can use applications on the phone that have friendly platforms. These 
apps can have either the option of buttons that will operate various parameters in the 
show, or the option of sliders, which will allow us to operate parameters at a much 
more dynamic level. 

To be able to integrate such interactions into the show we will need to have 
various applications installed on the computer that operate these signals and transmit 
them to various parameters. The computer will be the one that will facilitate the 
network connection for the projection or sound present in the show. In the case of 
lights there will be the option to operate them directly via MIDI signals, many of the 
light operating interfaces have such a connection available. 


Conclusions 

The show through the use of technology can move towards being transformed 
for a totally different experience. The spectator becomes an integral part of the 
performance, and the interaction between actors and spectators can be playful or 
perhaps even confrontational. 

In this kind of convention actors must arm themselves with a different kind of 
availability. It is very difficult to simulate in advance what is going to happen when 
the spectators intervene in the structure. Actors will enter into a convention that will 
contain a large dose of improvisation. 
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In addition to the availability part they will have to unpack the technical part. 
Most of the time it is essential to find people specialized on this side, but there are 
many more simplified variations that can be applied in just by viewing and 
understanding a few tutorials. In this case the actors will have to acquire the necessary 
knowledge to operate the platforms involved in the project, but this will add to the 
convention. The fact that the actor facilitates the experience himself and does not need 
an external element (a technician) can create a different relationship with the viewer. 

Integrating technology into the performance also brings an aesthetic change. 
The simple placement of various devices in the performance/meeting space is a 
scenography. Even more so when the spectator operates the various devices, the 
aesthetics of the dynamic will change. The spectator would not simply be a fixed 
element on a chair. He will move to operate one apparatus or another or sit in the 
comfort of his chair, but while using his mobile phone to change parameters in the 
performance. 
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